---
title: "Online Appendix for Correcting Bias in Perceptions of Public Opinion"
author:
- Joshua Kalla^[Yale University, Departments of Political Science and Statistics & Data Science, josh.kalla@yale.edu]
- Ethan Porter^[George Washington University, School of Media and Public Affairs, evporter@gwu.edu]
header-includes:
   - \usepackage{pdfpages}
output: 
  pdf_document: 
    keep_tex: yes
toc: true
numbersections: true
---

```{r, echo=FALSE, message=FALSE}
setwd("")
data.access <- read.csv("access_data.csv", stringsAsFactors = FALSE)
data.long <- read.csv("long_data.csv", stringsAsFactors = FALSE)
data.access.2018 <- read.csv("access_data_2018.csv", stringsAsFactors = FALSE)
data.long.2018 <- read.csv("long_data_2018.csv", stringsAsFactors = FALSE)
```

\textcolor{red}{Replication data and code can be accessed at https://doi.org/10.7910/DVN/0A1AM3.
A pre-analysis plan was filed at EGAP prior to conducting the experiment. See http://egap.org/registration/2872. The PAP is also appended to the end of this document.}

# Experimental Design

## Setting

Our field experiment was conducted from 10 October 2017 to 3 January 2018 among state legislators in the United States. We then replicated it from 24 September to 26 October 2018 among the legislators who were not compliers in the original study.

## Study Population

The starting universe for the experiment was all sitting United States state legislators in either the upper or lower chamber. We then limited the experiment to legislators we could identify as either Democrat or Republican (no independents were included), for whom we had contact information (email address and phone number), and for whom we could get 2016 presidential voting results. Election data were compiled from the Daily Kos Elections' 2016 presidential results for congressional and legislative districts (https://www.dailykos.com/stories/2017/2/6/1629653/-Daily-Kos-Elections-2016-presidential-results-for-congressional-and-legislative-districts). 

This led to a final universe of `r nrow(data.access)` state legislators. The below table provides summary statistics:

```{r, echo=FALSE, message=FALSE}
report.percent <- function(data, var) {
  df <- data[!is.na(data[,var]),]
  return(paste0(round(sum(df[,var]) / nrow(df), 3) * 100, "%"))
}
```

Variable | Freq.
---------| -----
Upper Chamber | `r report.percent(data.access, "upper")`
Democrat | `r report.percent(data.access, "democrat")`
Republican | `r report.percent(data.access, "republican")`
2016 Trump Percent | `r report.percent(data.access, "trump_percent")`
Median Household Income (log \$) | `r round(mean(data.access$log_medianhhincome), 1)`
Constituent Support (MRP): DREAM Act | `r report.percent(data.access, "est_immigr")`
Constituent Support (MRP): Mandatory Minimum | `r report.percent(data.access, "est_crime")`
Constituent Support (MRP): Renewable Fuels | `r report.percent(data.access, "est_enviro")`
Constituent Support (MRP): Background Checks | `r report.percent(data.access, "est_gun")`
Constituent Support (MRP): Minimum Wage | `r report.percent(data.access, "est_wage")`
Constituent Support (MRP): Highway Funding | `r report.percent(data.access, "est_transit")`
Constituent Support (MRP): Abortion | `r report.percent(data.access, "est_abortion")`
Constituent Support (MRP): ACA Repeal | `r report.percent(data.access, "est_aca")`

## Treatments

Treatments varied in two ways. First, state legislators could receive polling on 4 out of 8 possible policy issues. The issues were:

- Grant legal status to people who were brought to the US illegally as children, but who have graduated from a U.S. high school.
- Eliminate mandatory minimum sentences for non-violent drug offenders.
- Require a minimum amount of renewable fuels (wind, solar, and hydroelectric) in the generation of electricity even if electricity prices increase somewhat.
- Background checks for all sales, including at gun shows and over the Internet.
- Vote for Minimum wage: federal minimum wage to $12 an hour by 2020.
- Vote for Highway and Transportation Funding Act: Authorizes $305 Billion to repair and expand highways, bridges, and transit over the next 5 years.
- Make abortions illegal in all circumstances.
- Vote for Repeal Affordable Care Act: Would repeal the Affordable Care Act of 2009 (also known as Obamacare).

Note that first six issues were framed on the liberal side and the final two were framed on the conservative side. These issues were selected to include a wide range of policy issues across a number of domains, with an emphasis for those on which state legislators might be expected to hold opinions because they influence state public policy. We were also constrained by the questions asked in the 2016 Cooperative Congressional Election Study (CCES). 

In a below section, we discuss how we generated the district-specific polling from the 2016 CCES using a multilevel regression and poststratification (MRP) procedure similar to that used by Ahler and Broockman (2017) and Broockman and Skovron (2018). We generated the regional polling by taking the weighted average within Census region using the Common Content weights supplied with the CCES.  

Second, state legislators were provided either with a regional polling aggregate (a placebo condition) or their district-specific polling. 

The below images show what the treatments looked like. 

### Home Screen

\includegraphics{"DP_home"}

### Login Screen (to identify compliers)

\includegraphics{"DP_login"}

### Regional Polling

\includegraphics{"DP_placebo"}

### District Specific Polling

\includegraphics{"DP_treat"}

## Randomization Procedure

Our randomization procedure followed one similar to that used in Green et al. (2017). In conducting their experiment, they were interested in measuring the effect of three distinct treatments on three distinct policies: reducing intimate partner violence, reducing the stigma surrounding abortions, and reducing teacher absenteeism. By randomly assigning villages to receive a subset of these treatments or a placebo and by collecting outcome measures for all policies regardless of treatment assignment, Green et al. are able to use the treatment group for one policy domain as the control group for the other and vice-versa. Doing so substantially increases the statistical power of the experiment by allowing for both within-village comparisons across policies as well as between-villages. 

Our experiment benefited from a similar design. In our case, legislators were randomly assigned to receive polling information on 4 out of a possible 8 policy issues. Furthermore, half of legislators were randomly assigned to receive district-specific polling information (treatment) while half were randomly assigned to receive regional polling aggregates (placebo). 

To conduct the randomization, the below procedure was followed:

1. State legislators were organized into blocks based on their party (Democrat or Republican, no independents were included) and the number of issues they were out of sync on (0 or 1, 2-5, or 6-8). On each issue, a legislator was coded as out of sync if the MRP estimates predicted that a majority of constituents supported the liberal (conservative) position but the legislator was a Republican (Democrat). While this is likely a noisy measure of dyadic representation, as Higgins et al. (2016, p.7375) note, "even when the covariates contain no information about the outcomes, blocking cannot increase the variance of the treatment effect estimator compared to when no blocking is done". 
2. Within blocks, half of state legislators were randomly assigned to receive district-specific polling and half were randomly assigned to receive regional aggregate polling.
3. Within blocks, state legislators were randomly assigned to receive polling on one of 70 combinations (8 choose 4 = 70) of issues. Of 8 possible issues, legislators were randomly assigned to receive information on 4. Whether the information was district-specific or regional aggregate polling depended on their random assignment from #2. 
4. State legislators were invited via email and phone call to access District Pulse and their polling information.
5. District Pulse logs which state legislators access the password-protected website, allowing us to identify the compliers.
6. State legislators were invited via email and phone call to a post-treatment survey. Note that the post-treatment survey was conducted by a researcher unaffiliated with the treatment and District Pulse.
7. Study 2 maintained the same random assignment as the original study.

## Balance Check

In the below tables, we check whether pre-treatment covariates are predictive of treatment. The first table looks at all state legislators, the second table is subsetted to the compliers (those who access the polling information), and the third table is subsetted to the compliers who also responded to the post-treatment survey. In each table, the first column is whether the state legislator was randomly assigned to receive district-specific or regional polling. The remaining 8 columns are whether the legislator received polling on that particular issue, regardless of the geography of the polling. 

As the tables show, pre-treatment covariates are not consistently predictive of treatment assignment, suggesting: (1) the randomization was properly implemented, (2) treatment did not influence who accessed the polling information, and (3) treatment did not influence who completed the post-treatment survey.

First is the balance check for the original study.

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
require(stargazer)
lm1 <- lm(treat_district ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm2 <- lm(treat_immigr ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm3 <- lm(treat_crime ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm4 <- lm(treat_enviro ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm5 <- lm(treat_gun ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm6 <- lm(treat_wage ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm7 <- lm(treat_transit ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm8 <- lm(treat_abortion ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
lm9 <- lm(treat_aca ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access)
stargazer(lm1, lm2, lm3, lm4, lm5, lm6, lm7, lm8, lm9, title="Balance Check: All Legislators", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          dep.var.labels = c("District", "DREAM", "Man. Min.", "Renewables", "Gun", "Wage", "Transit", "Abortion", "ACA"),
          covariate.labels = c("\\# Out Sync", "Trump \\%", "Democrat", "HH Income"),
          column.sep.width = "1pt")
```

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
lm1.compliers <- lm(treat_district ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm2.compliers <- lm(treat_immigr ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm3.compliers <- lm(treat_crime ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm4.compliers <- lm(treat_enviro ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm5.compliers <- lm(treat_gun ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm6.compliers <- lm(treat_wage ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm7.compliers <- lm(treat_transit ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm8.compliers <- lm(treat_abortion ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
lm9.compliers <- lm(treat_aca ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1))
stargazer(lm1.compliers, lm2.compliers, lm3.compliers, lm4.compliers, lm5.compliers, lm6.compliers, lm7.compliers, lm8.compliers, lm9.compliers, title="Balance Check: Among Those Who Accessed Website", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          dep.var.labels = c("District", "DREAM", "Man. Min.", "Renewables", "Gun", "Wage", "Transit", "Abortion", "ACA"),
          covariate.labels = c("\\# Out Sync", "Trump \\%", "Democrat", "HH Income"),
          column.sep.width = "1pt")
```

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
lm1.compliers.survey <- lm(treat_district ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm2.compliers.survey <- lm(treat_immigr ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm3.compliers.survey <- lm(treat_crime ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm4.compliers.survey <- lm(treat_enviro ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm5.compliers.survey <- lm(treat_gun ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm6.compliers.survey <- lm(treat_wage ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm7.compliers.survey <- lm(treat_transit ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm8.compliers.survey <- lm(treat_abortion ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
lm9.compliers.survey <- lm(treat_aca ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))
stargazer(lm1.compliers.survey, lm2.compliers.survey, lm3.compliers.survey, lm4.compliers.survey, lm5.compliers.survey, lm6.compliers.survey, lm7.compliers.survey, lm8.compliers.survey, lm9.compliers.survey, title="Balance Check: Among Those Who Accessed Website and Took Post-Treatment Survey", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          dep.var.labels = c("District", "DREAM", "Man. Min.", "Renewables", "Gun", "Wage", "Transit", "Abortion", "ACA"),
          covariate.labels = c("\\# Out Sync", "Trump \\%", "Democrat", "HH Income"),
          column.sep.width = "1pt")
```

\newpage
Here is the balance check for the second study. We do not have a balance check for those legislators who accessed the website and took the post-treatment survey because there were only 4.

```{r, echo=FALSE, message=FALSE, results='asis'}
require(stargazer)
lm1 <- lm(treat_district ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm2 <- lm(treat_immigr ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm3 <- lm(treat_crime ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm4 <- lm(treat_enviro ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm5 <- lm(treat_gun ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm6 <- lm(treat_wage ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm7 <- lm(treat_transit ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm8 <- lm(treat_abortion ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
lm9 <- lm(treat_aca ~ out_total + trump_percent + democrat + log_medianhhincome, data = data.access.2018)
stargazer(lm1, lm2, lm3, lm4, lm5, lm6, lm7, lm8, lm9, title="Balance Check: All Legislators", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          dep.var.labels = c("District", "DREAM", "Man. Min.", "Renewables", "Gun", "Wage", "Transit", "Abortion", "ACA"),
          covariate.labels = c("\\# Out Sync", "Trump \\%", "Democrat", "HH Income"),
          column.sep.width = "1pt")
```

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
lm1.compliers <- lm(treat_district ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm2.compliers <- lm(treat_immigr ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm3.compliers <- lm(treat_crime ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm4.compliers <- lm(treat_enviro ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm5.compliers <- lm(treat_gun ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm6.compliers <- lm(treat_wage ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm7.compliers <- lm(treat_transit ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm8.compliers <- lm(treat_abortion ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
lm9.compliers <- lm(treat_aca ~ out_total + trump_percent + democrat + log_medianhhincome, data = subset(data.access.2018, data.access.2018$accessed_site == 1))
stargazer(lm1.compliers, lm2.compliers, lm3.compliers, lm4.compliers, lm5.compliers, lm6.compliers, lm7.compliers, lm8.compliers, lm9.compliers, title="Balance Check: Among Those Who Accessed Website", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          dep.var.labels = c("District", "DREAM", "Man. Min.", "Renewables", "Gun", "Wage", "Transit", "Abortion", "ACA"),
          covariate.labels = c("\\# Out Sync", "Trump \\%", "Democrat", "HH Income"),
          column.sep.width = "1pt")
```

## MRP Procedure

Our MRP procedure follows closely from that used by Ahler and Broockman (2017) and Broockman and Skovron (2018) and we thank the authors for sharing their code.

Estimation of our MRP model proceeded in two stages. First, a hierarchical logistic choice model was estimated for each public opinion survey question studied. Our models include predictors at three different levels. At the individual level, we include random effects for the respondent’s education, gender, and race/ethnicity as well as interaction effects between gender/race and education/race. At the state-house and -senate district level, we include individual district random effects, fixed effects for the districts’ median household income, and Trump’s share of the 2016 Presidential vote in the district. State random effects, centered around regional random effects, were included in  the individual model as well. This model yields predictions for the share of individuals in any given state legislative district who support the various outcome measures in all possible combinations of race, gender, and education.

The final step in constructing district-level estimates is poststratification. We first use data from the US Census's American Community Survey 2015 5-Year file to calculate the share of individuals in each state legislative district that fall into each 'cell' (e.g., what share of individuals living in a particular state legislative district are white college-educated women?).

We then merge these cell-level district proportion estimates from the Census with our cell-level opinion estimates from the multilevel regression model to construct the district-level opinion estimates. This poststratification process is a straightforward aggregation process by which estimates for each cell in each district are summed in proportion to the share of the district that they represent.

The result of this poststratification process are estimates of district support for each issue for each of the nation’s state legislative districts.

## Outcome Measures

Our post-treatment survey asked three questions:

1. Please indicate whether you agree or disagree with the following policy positions: Agree vs. Disagree binary response.
2. Now imagine you had to vote on the following policy proposals. How would you vote? Vote Yes vs. Vote No binary response.
3. Consider the people living in your legislative district. To the best of your knowledge, what percentage of these people do you think would agree with the following policy statements? The slider below goes from 0% (no one) to 100% (everyone). Please give your best guess by clicking in the slider. Slider response option.

For each of the above questions, the following 8 statements were presented in a **random** order:

- Grant legal status to people who were brought to the US illegally as children, but who have graduated from a U.S. high school.
- Eliminate mandatory minimum sentences for non-violent drug offenders.
- Require a minimum amount of renewable fuels (wind, solar, and hydroelectric) in the generation of electricity even if electricity prices increase somewhat.
- Background checks for all gun sales, including at gun shows and over the Internet.
- Increase the federal minimum wage to $12 an hour by 2020.
- Authorize $305 Billion to repair and expand highways, bridges, and transit over the next 5 years.
- Make abortions illegal in all circumstances.
- Repeal the Affordable Care Act of 2009 (also known as Obamacare)

## Survey Response Rates and Characteristics of Respondents

Our survey response rate in the original study was `r round(nrow(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1)) / nrow(subset(data.access, data.access$accessed_site == 1)) * 100, 1)`%. This is calculated by taking the number of compliers and responders divided by the number of compliers. Following the design in Broockman et al. (2017a), we only re-survey the compliers for the purpose of the experiment. This survey response rate exceeds the rates found in recent surveys of political elites (cf. Broockman and Skovron 2018, Broockman et al. 2017b, Teele et a. 2017). For our follow-up study, our survey response rate was `r round(nrow(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1)) / nrow(subset(data.access.2018, data.access.2018$accessed_site == 1)) * 100, 1)`%

The below table shows demographic characteristics of state legislators at the three stages of the experiment for the original study. The first column is for all state legislators who were invited to access polling data. The second column (compliers) are those state legislators who accessed the polling data. The third column (compliers + responders) are those state legislators who both accessed the polling data and responded to our post-treatment survey. It is among this set of state legislators for whom we can estimate treatment effects. Notably, this subset of state legislators is more Democratic than the overall universe of state legislators, as we discuss elsewhere.

Variable | Starting Universe | Compliers | Compliers + Responders
---------| ----- |  ----- |  ----- 
Upper Chamber | `r report.percent(data.access, "upper")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "upper")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "upper")`
Democrat | `r report.percent(data.access, "democrat")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "democrat")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "democrat")`
Republican | `r report.percent(data.access, "republican")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "republican")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "republican")`
2016 Trump Percent | `r report.percent(data.access, "trump_percent")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "trump_percent")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "trump_percent")`
Median Household Income (log \$) | `r round(mean(data.access$log_medianhhincome, na.rm = TRUE), 1)` | `r round(mean(data.access[data.access$accessed_site == 1, "log_medianhhincome"], na.rm = TRUE), 1)` | `r round(mean(data.access[data.access$accessed_site == 1 & data.access$t1_respondent == 1, "log_medianhhincome"], na.rm = TRUE), 1)`
Constituent Support (MRP): DREAM Act | `r report.percent(data.access, "est_immigr")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_immigr")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_immigr")`
Constituent Support (MRP): Mandatory Minimum | `r report.percent(data.access, "est_crime")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_crime")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_crime")`
Constituent Support (MRP): Renewable Fuels | `r report.percent(data.access, "est_enviro")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_enviro")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_enviro")`
Constituent Support (MRP): Background Checks | `r report.percent(data.access, "est_gun")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_gun")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_gun")`
Constituent Support (MRP): Minimum Wage | `r report.percent(data.access, "est_wage")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_wage")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_wage")`
Constituent Support (MRP): Highway Funding | `r report.percent(data.access, "est_transit")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_transit")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_transit")`
Constituent Support (MRP): Abortion | `r report.percent(data.access, "est_abortion")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_abortion")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_abortion")`
Constituent Support (MRP): ACA Repeal | `r report.percent(data.access, "est_aca")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "est_aca")` | `r report.percent(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1), "est_aca")`
N | `r nrow(data.access)` | `r nrow(subset(data.access, data.access$accessed_site == 1))` | `r nrow(subset(data.access, data.access$accessed_site == 1 & data.access$t1_respondent == 1))`

The below table is the same, but for the follow-up study.

Variable | Starting Universe | Compliers | Compliers + Responders
---------| ----- |  ----- |  ----- 
  Upper Chamber | `r report.percent(data.access.2018, "upper")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "upper")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "upper")`
Democrat | `r report.percent(data.access.2018, "democrat")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "democrat")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "democrat")`
Republican | `r report.percent(data.access.2018, "republican")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "republican")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "republican")`
2016 Trump Percent | `r report.percent(data.access.2018, "trump_percent")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "trump_percent")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "trump_percent")`
Median Household Income (log \$) | `r round(mean(data.access.2018$log_medianhhincome, na.rm = TRUE), 1)` | `r round(mean(data.access.2018[data.access.2018$accessed_site == 1, "log_medianhhincome"], na.rm = TRUE), 1)` | `r round(mean(data.access.2018[data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1, "log_medianhhincome"], na.rm = TRUE), 1)`
Constituent Support (MRP): DREAM Act | `r report.percent(data.access.2018, "est_immigr")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_immigr")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_immigr")`
Constituent Support (MRP): Mandatory Minimum | `r report.percent(data.access.2018, "est_crime")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_crime")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_crime")`
Constituent Support (MRP): Renewable Fuels | `r report.percent(data.access.2018, "est_enviro")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_enviro")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_enviro")`
Constituent Support (MRP): Background Checks | `r report.percent(data.access.2018, "est_gun")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_gun")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_gun")`
Constituent Support (MRP): Minimum Wage | `r report.percent(data.access.2018, "est_wage")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_wage")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_wage")`
Constituent Support (MRP): Highway Funding | `r report.percent(data.access.2018, "est_transit")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_transit")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_transit")`
Constituent Support (MRP): Abortion | `r report.percent(data.access.2018, "est_abortion")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_abortion")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_abortion")`
Constituent Support (MRP): ACA Repeal | `r report.percent(data.access.2018, "est_aca")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1), "est_aca")` | `r report.percent(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1), "est_aca")`
N | `r nrow(data.access.2018)` | `r nrow(subset(data.access.2018, data.access.2018$accessed_site == 1))` | `r nrow(subset(data.access.2018, data.access.2018$accessed_site == 1 & data.access.2018$t1_respondent == 1))`

## Implementation Procedures

### Invitation to District Pulse

The first two emails were sent using Yet Another Mail for GMail and was sent from an official "**REMOVED**\@districtpulse.us" email account. Because of sending limits withing GMail, these emails were spread across multiple days. The order of send was randomly assigned. The third email was sent using MailChimp from an official "**REMOVED**\@districtpulse.us" email account. The MailChimp sending limit was sufficiently high to allow for all emails to send in one day. 

Emails were sent on 10-12 October, 17-19 October, and 25 October. Phone calls were conducted by undergraduate research assistants from 12 October - 1 November.

SUBJECT: Access detailed polling
Below is the email: 

Dear Assembly Member **NAME**,
I am **REMOVED** and I am a Professor at **REMOVED** University. I recently emailed to invite you to access new detailed policy polling on what Americans think about some of the most important issues facing our country.

I understand the difficulty in finding high-quality polling information on what Americans think about various public policies. That's why I’ve collected information from the Cooperative Congressional Election Study — a 65,000 person survey funded by the National Science Foundation — into easily digestible maps so that elected officials like yourself can learn where the public stands on various policies.

Accessing this polling is totally free. As an academic who studies public opinion and relies on the taxpayer-funded National Science Foundation, I want to return the favor by providing this information to elected officials across the country.

To access polling information tailored specifically for you, please visit **URL**. Your password is **PW**.

If you have any trouble accessing your polling information or would like to learn more, please do not hesitate to get in touch with me at **EMAIL**.
 
Sincerely,

**REMOVED**

Below is the phone script:

Hi, my name is **RA NAME** and I am a research assistant at District Pulse. District Pulse uses data from the Cooperative Congressional Election Study to create a free dashboard for high-quality polling on the issues and policies people care most about. We are providing free access to state legislators and want to make sure you see this. Your office can log in by going to **URL** and entering password **PW**. Do you want to try logging in right now? I can help you if you have any issues. Thank you very much and have a great day!

### Invitation to District Pulse - 2018 Electoral Version

SUBJECT: Access detailed polling before the elections
Below is the email: 

Dear **NAME**,

I am **REMOVED** and I am an Assistant Professor at **REMOVED**. In advance of this fall's elections, I want to invite you to access new detailed polling on what Americans think about some of the most important issues facing our country.

In every election year, it's difficult to find high-quality polling information on what Americans think about the issues most important to them. That's why I've collected information from the Cooperative Congressional Election Study - a 65,000 person survey funded by the National Science Foundation - into easily digestible maps so that elected officials like yourself can learn where the public stands on various policies.

This is especially important information during an election year. 

Accessing this polling is totally free. As an academic who studies public opinion and relies on the taxpayer-funded National Science Foundation, I want to return the favor by providing this information to elected officials across the country.

To access polling information tailored specifically for you, please visit **URL**. Your password is **PW.** 

If you have any trouble accessing your polling information or would like to learn more, please do not hesitate to get in touch with me at **EMAIL**.


Sincerely,
**NAME**
Assistant Professor, **UNIVERSITY**

### Invitation to Post-Treatment Survey

Note that the post-treatment survey was conducted by a researcher unaffiliated with the treatment and District Pulse.

Emails were sent on 13 November, 16 November, 20 November, 28 November, 1 December, 7 December, and 18 December. Phone calls were conducted by undergraduate research assistants from 28 November - 8 December. 

Below is the email: 

Dear Assembly Member **NAME**,

As an elected official in the United States, you have been selected to participate in the 2017 National Survey of American Politicians, a brief, five-minute survey that will help scholars better understand how elected officials have achieved their positions in politics. Hundreds of officials around the world have already answered these questions to help scholars understand electoral success and legislative effectiveness. Please join them in answering this short survey.

Please follow this link to the Survey: **URL**

All survey answers will be completely anonymous and confidential; neither your name or any identifying information will be made available to anyone at any time.

Please follow the link to participate: **URL**

Or copy and paste the URL below into your internet browser: **URL**

Thank you very much for taking your valuable time to help with this study! If you have any questions about the survey, please do not hesitate to contact us at **EMAIL**.

Sincerely,
**REMOVED**

Below is the phone script:

Hi, my name is **RA NAME** and I am a research assistant for Professor **REMOVED** at **REMOVED** University. Your office has been selected to participate in the 2017 National Survey of American Politicians. This is a brief online survey that **LEGISLATOR NAME** or the legislative staff can complete. Is there a good email for me to send the survey? Great! I’ll send you the survey right now. Thank you very much and have a great day!

Upon accessing the survey, respondents saw the following welcome page:

**2017 National Survey of American Politicians**

The National Survey of American Politicians is an **independent, confidential research study** of the experiences and views of of the remarkable people who serve in public office in the United States. The survey is conducted by researchers from leading universities around the country. The current survey is administered by Professor **REMOVED** of **REMOVED** University. It should take less than ten minutes to complete. 

The survey is voluntary and completely anonymous. Your name will not be made available to anyone. If you have any questions regarding this research, you can email Professor **REMOVED** at **REMOVED**.edu.

Note that in 2018, the same text was removed, but the survey was called the 2018 National Survey of American Politicians.


## Flow Diagram of Experiment Implementation and Sample Size

Below we document each step of the experiment and the corresponding sample size.

\newpage

### 2017 Study

Here is the flow diagram from the 2017 study.

<br>
<center>
![](flow_2017.png){width=450px}
</center>

<br>

\newpage
### 2018 Study

Here is the flow diagram from the 2018 replication study.

<center>
![](flow_2018.png){width=450px}
</center>
\newpage

# District Pulse Validation

District Pulse was designed by Chi/Donahoe, a digital creative consultant firm specializing in data visualizations and interactive experiences for political clients, such as the AFL-CIO, the Center for American Progress, and the Elizabeth Warren for Senate campaign. 

Before launching District Pulse, we pre-tested its content by sending the introductory email along with a URL and password to two former legislative assistants and one sitting state legislator (who was excluded from the experiment). We simply asked them "What do you think?" in brief interviews conducted over Facebook Messenger. 

A former Legislative Aide for Colorado House Democrats and the Oklahoma Senate replied: "I would have logged in, vetted it, then seeing all signs good - passed it along to the rep. This was always a resource being requested in office. And some lobbyists carried binders of confidential polling info. The information on the website is all written clearly."

A former Staff Assistant in the US Congress replied: "i think it's good. there is a little bit of a disconnect for me between the text talking about the polling in an individual district and then the map below being divided by regions. but that isn't a huge deal." When asked to say more about the disconnect, the staffer replied "so as a staffer, i cared about polling in the district and polling in the state. i don't recall looking at regional data very often. i think it might make it easier for people to ignore that info." The staffer confirmed that they understood the district-specific polling information and would have ignored the regional polling.

A sitting state legislator replied: "This strikes me as an exciting email to receive! I don't have any constructive feedback. It reads really well."

In addition, we collected any replies received from legislators or their staff after they were invited to access District Pulse. We only received 16 replies. The replies consistently express gratitude for this type of polling information. Replies from legislators in the district specific polling condition suggest that the MRP estimates are generally consistent with that the legislators expect based on their own polling. The replies are listed below, slightly modified to maintain anonymity:

- Thanks for sending along that link! It’s interesting to see what our constituency thinks about these issues.
- Thank you - this is very interesting! What was the date of the survey and how were the questions worded? I ask because the health care answer stands out to me (your national numbers not just my district). Recent polls show that less than a quarter of people think the ACA should be repealed, but your numbers show around 50% nationally, and around 40% for my district. I am just trying to figure out why the drastic difference. The timing and wording might explain it.
- Thank you for sharing
- Thanks for sharing this information. Could you say a bit more about the distribution of the 65,000 N across state legislative districts or at least across states? Using 5411 lower chamber legislative districts (per NCSL) a perfectly even distribution (which I'm sure is not the case) is about 12, which I know can't be right. So I'm sure there is some other methodology used to render the district-level data. Is the survey designed with a state-level valid N which is then extrapolated to the district level based on the extent to which demographic sub-samples at the district level vary from the demographic sub-samples at the state level?
- Thanks. I will try to Access information through the link.
- I got the invitation to check out polling information by District Pulse, and did so - I was particularly interested in the statistic on % support for renewable energy even if prices increase. I was wondering, do you have a way I could access that percent district-by-district for the other districts in MA besides my own?
- Thank you so much for offering me the opportunity to learn more about my district as it pertains to important issues facing our constituents in the ##th Legislative District. I look forward to reviewing the policy polling.
- Thank you for giving me access to this data set. There are clear parallels between the findings of the CCES and what I know of the sentiments of the people in my district. Some of the values seem a little high but on target. This will be a good tool to help me frame my survey questions and interactions with my constituents going forward.
- Assemblywoman XX asked me to find out whether you have access to polling data that would be specific to her district rather than regional data. Are we not understanding how to use the website or is the data regional? Thank you for your help. **NOTE: In the regional condition.**
- Thank you for the information. I reviewed it and have found it incorrect. I have done in-depth polling in my district of most likely voters...my data is very diffferent on every issue than yours. Thank you. **NOTE: In the regional condition.**
- Thank you for sharing the polling information you have been working on nationwide, it is very much appreciated. It is helpful to have an understanding of what folks would like to see done, how needs may have changed, perceptions of how things in general are going and if there are potential gaps in resources or services that are important to them. Thank you again. Have a great day and a good week!
- Thank you for sending Representative XXX a link to polling information from constituents across District ##. She is currently away on maternity leave but wanted me to reach out and express our gratitude for this valuable data.
- Thank for making this available.
- Thanks for this info. I'll try to look into it soon.
- Thank you for this information.
- wanted to check with you to see if we could get info. on the below just for Representative XX's district. **NOTE: In the regional condition.**

# Who Accessed District Pulse?

Overall, `r sum(data.access$accessed_site)` state legislators out of `r nrow(data.access)` or `r round(sum(data.access$accessed_site) / nrow(data.access), 2)*100`% accessed District Pulse at least once during the original study. `r sum(data.access.2018$accessed_site)` state legislators out of `r nrow(data.access.2018)` or `r round(sum(data.access.2018$accessed_site) / nrow(data.access.2018), 2)*100`% accessed District Pulse at least once during the follow-up study. We were able to track access by logging when a password was entered. 

In this section, we report which variables predict access, examining only the original study due to the smaller sample size of the follow-up study. The below table regresses a binary variable for whether or not the polling data was accessed on a series of predictors. The predictors are whether the legislator is a Democrat, Trump's 2016 presidential vote share in the district, whether the legislator sits in the upper chamber, the log of the media household income in the district, an indicator for whether an RA called the legislative office to inform them of District Pulse (in addition to just the emails; note that the order in which legislators was called was randomly assigned), and state fixed effects. 

Across the various model specifications, two types of variables are consistently significant predictors. First is the politics of the district. Being a Democrat and Trump's 2016 presidential performance are both significant predictors of accessing the website. Second are the economics of the district. Legislators from districts with higher median household incomes are also more likely to have accessed the website. These findings are also reflected in the table included in "Characteristics of Respondents", the complier column. 

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
acc1 <- lm(accessed_site ~ democrat + upper + log_medianhhincome, data = data.access)
acc1.s <- lm(accessed_site ~ democrat + upper + log_medianhhincome + as.factor(state), data = data.access)
acc2 <- lm(accessed_site ~ trump_percent + upper + log_medianhhincome, data = data.access)
acc2.s <- lm(accessed_site ~ trump_percent + upper + log_medianhhincome + as.factor(state), data = data.access)
acc3 <- lm(accessed_site ~ democrat + trump_percent + upper + log_medianhhincome, data = data.access)
acc3.s <- lm(accessed_site ~ democrat + trump_percent + upper + log_medianhhincome + as.factor(state), data = data.access)
acc4 <- lm(accessed_site ~ called_districtpulse, data = data.access)
acc4.s <- lm(accessed_site ~ called_districtpulse + as.factor(state), data = data.access)
acc5 <- lm(accessed_site ~ democrat + trump_percent + upper + log_medianhhincome + called_districtpulse, data = data.access)
acc5.s <- lm(accessed_site ~ democrat + trump_percent + upper + log_medianhhincome + called_districtpulse + as.factor(state), data = data.access)
stargazer(acc1, acc1.s, acc2, acc2.s, acc3, acc3.s, acc4, acc4.s, acc5, acc5.s, title="Predictors of Accessing District Pulse", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Democrat", "Trump \\%", "Upper Chamber", "HH Income", "Called by RA"),
          column.sep.width = "0pt",
          dep.var.caption="Accessed Polling Data",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", rep(c("No", "Yes"), 5))),
          digits = 2)
```

# Experimental Results

In analyzing the experiment for the original study, we first limit our data to the compliers (those who accessed the polling data) who responded to the post-treatment survey. This results in `r nrow(subset(data.access, data.access$t1_respondent == 1 & data.access$accessed_site == 1))` subjects. We then transformed this dataset into a "long" dataset where each row is a distinct policy. Since there are 8 possible policy areas, this dataset has `r nrow(subset(data.access, data.access$t1_respondent == 1 & data.access$accessed_site == 1))` * 8 = `r nrow(data.long)`. Each row then has three columns for the different dependent variables: personal policy agreement, expected voting behavior, and perception of constituent support. However, not every respondent answered every question. This non-response leaves `r sum(!is.na(data.long$dv_policy))` observations for the personal policy agreement DV, `r sum(!is.na(data.long$dv_vote))` for the expected voting behavior DV, and `r sum(!is.na(data.long$dv_const))` for the perception of constituent support DV. 

According to our pre-analysis plan, the following rules were used in defining the outcome measures:

- **Personal Policy Agreement**: We will create an outcome of policy agreement defined as 1 for agreement and 0 for disagreement. For each issue-legislator combination, we will exclude those where support/opposition is within five percentage points of 50% (i.e., for issues where district opinion is between 45-55%) or who did not answer this question.
- **Expected Voting Behavior**: We will create an outcome of voting agreement defined as 1 if a majority of respondents in the district SUPPORT the proposal and the legislator votes YES. 1 if a majority of respondents in the district OPPOSE the proposal and the legislator votes NO. 0 if a majority of respondents in the district SUPPORT the proposal and the legislator votes NO. 0 if a majority of respondents in the district OPPOSE the proposal and the legislator votes YES. Again, for each issue-legislator combination, we will exclude those where support/opposition is within five percentage points of 50% (i.e., for issues where district opinion is between 45-55%) or who did not answer this question.
- **Perception of Constituent Support**: We will create an outcome of accuracy defined as the absolute value of the legislator’s response minus district support on the issue calculated using MRP. 

A final column of note in this long dataset is a treatment indicator. The treatment is coded as 0 if the state legislator received no polling information on this issue; 1 if the state legislator received regional aggregate polling on this issuel 2 if the state legislator received district-specific polling on this issue.

We analyzed the data by regressing the dependent variable on the treatment indicator using OLS and clustering the standard errors at the state legislator level. In some models, we include the pre-treatment covariates indicators for whether the legislator is a Democrat or Republican, indicator for upper vs lower chamber, 2016 Trump vote share in the district, and state fixed effects. These were all specified in our pre-analysis plan. Note that no baseline survey was conducted, so no baseline responses are included as covariates. 

The below tables summarize the results. 

First, we present mean values of each of the three outcomes across the three treatments. This transparent analysis shows that the district-specific polling does not increase the likelihood that a state legislator will personally agree with the policy positions of a majority of her constituents, does not increase the likelihood that a state legislator's expected voting behavior on a policy position will align with the policy positions of a majority of her constituents, and does not increase the likelihood that a state legislator accurately perceives the policy positions of her constituents.

Treatment | Avg. Policy Agreement | Avg. Voting Behavior | Avg. Perception
--------- |--------- |--------- |---------
Control | `r round(mean(data.long[data.long$treat == 0, "dv_policy"], na.rm = TRUE), 2)` | `r round(mean(data.long[data.long$treat == 0, "dv_vote"], na.rm = TRUE), 2)` | `r round(mean(data.long[data.long$treat == 0, "dv_const"], na.rm = TRUE), 2)`
Regional | `r round(mean(data.long[data.long$treat == 1, "dv_policy"], na.rm = TRUE), 2)`  | `r round(mean(data.long[data.long$treat == 1, "dv_vote"], na.rm = TRUE), 2)` | `r round(mean(data.long[data.long$treat == 1, "dv_const"], na.rm = TRUE), 2)`
District | `r round(mean(data.long[data.long$treat == 2, "dv_policy"], na.rm = TRUE), 2)`  | `r round(mean(data.long[data.long$treat == 2, "dv_vote"], na.rm = TRUE), 2)` | `r round(mean(data.long[data.long$treat == 2, "dv_const"], na.rm = TRUE), 2)`

The next three tables look at the effects on policy agreement, voting behavior, and constituent perception using OLS with cluster-robust standard errors at the state legislator level. In each table, the first column regresses the outcome on just the treatment indicator. In the second column, state fixed effects are added. In the third column, the pre-treatment covariates specified in the pre-analysis plan are added without state fixed effects. In the fourth column, the pre-treatment covariates specified in the pre-analysis plan are added with state fixed effects. The fifth column uses the same specification and includes data from the follow-up study. 

\newpage
This first table shows the non-effects of district-specific polling on policy agreement.

```{r, echo=FALSE, message=FALSE, results='asis'}
#Cluster SEs, https://economictheoryblog.com/2016/12/13/clustered-standard-errors-in-r/
# load necessary packages for importing the function
library(RCurl)
# import the function from repository
url_robust <- "https://raw.githubusercontent.com/IsidoreBeautrelet/economictheoryblog/master/robust_summary.R"
eval(parse(text = getURL(url_robust, ssl.verifypeer = FALSE)),
     envir=.GlobalEnv)

library(dplyr)
data_long_both <- bind_rows(data.long, data.long.2018)

policy1 <- lm(dv_policy ~ as.factor(treat), data = data.long)
policy1.s <- lm(dv_policy ~ as.factor(treat) + as.factor(state), data = data.long)
policy2 <- lm(dv_policy ~ as.factor(treat) + democrat + upper + trump_percent, data = data.long)
policy2.s <- lm(dv_policy ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.long)
policy3 <- lm(dv_policy ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data_long_both)
stargazer(policy1, policy1.s, policy2, policy2.s, policy3, title="Experimental Results: Personal Policy Agreement", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Personal Policy Agreement",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", c(rep(c("No", "Yes"), 2), "Yes"))),
          digits = 2,
          se=list(coef(summary(policy1,cluster = c("anonid")))[, 2],
                  coef(summary(policy1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(policy2,cluster = c("anonid")))[, 2],
                  coef(summary(policy2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(policy3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage
This second table shows the non-effects of district-specific polling on expected voting behavior.

```{r, echo=FALSE, message=FALSE, results='asis'}
vote1 <- lm(dv_vote ~ as.factor(treat), data = data.long)
vote1.s <- lm(dv_vote ~ as.factor(treat) + as.factor(state), data = data.long)
vote2 <- lm(dv_vote ~ as.factor(treat) + democrat + upper + trump_percent, data = data.long)
vote2.s <- lm(dv_vote ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.long)
vote3 <- lm(dv_vote ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data_long_both)
stargazer(vote1, vote1.s, vote2, vote2.s, vote3, title="Experimental Results: Expected Voting Behavior", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Expected Voting Behavior",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", c(rep(c("No", "Yes"), 2), "Yes"))),
          digits = 2,
          se=list(coef(summary(vote1,cluster = c("anonid")))[, 2],
                  coef(summary(vote1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(vote2,cluster = c("anonid")))[, 2],
                  coef(summary(vote2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(vote3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage
This third table shows the non-effects of district-specific polling on perceptions of constituent support.

```{r, echo=FALSE, message=FALSE, results='asis'}
const1 <- lm(dv_const ~ as.factor(treat), data = data.long)
const1.s <- lm(dv_const ~ as.factor(treat) + as.factor(state), data = data.long)
const2 <- lm(dv_const ~ as.factor(treat) + democrat + upper + trump_percent, data = data.long)
const2.s <- lm(dv_const ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.long)
const3 <- lm(dv_const ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data_long_both)
stargazer(const1, const1.s, const2, const2.s, const3, title="Experimental Results: Perception of Constituent Support", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Perception of Constituent Support",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", c(rep(c("No", "Yes"), 2), "Yes"))),
          digits = 2,
          se=list(coef(summary(const1,cluster = c("anonid")))[, 2],
                  coef(summary(const1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(const2,cluster = c("anonid")))[, 2],
                  coef(summary(const2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(const3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage
## Potential Moderator: Electoral Safety
Based on suggestions from a reviewer, we also investigate whether electoral safety moderates treatment effects. It is possible that legislators in safe districts may have less of an incentive to learn about the polling than legislators in a more marginal district. We investigate this several ways in the below table. In the first column, we interact treatment assignment with the legislator's vote share in the previous election. In the second and third columns, we look at subgroups among legislators who won with less than or equal to 65% of the vote (Column Two) or more than 65% of the vote (Column Three). Overall, we find no significant evidence of an interaction between electoral safety and receiving treatment. However, we note that our limited sample size means that these estimates are imprecise and recommend further research into this question.

```{r, echo=FALSE, message=FALSE, results='asis'}
column1 <- lm(dv_const ~ as.factor(treat)*share, data = data.long)
column2 <- lm(dv_const ~ as.factor(treat), data = subset(data.long, share <= 0.65))
column3 <- lm(dv_const ~ as.factor(treat), data = subset(data.long, share > 0.65))

stargazer(column1, column2, column3, title="Experimental Results: Perception of Constituent Support Moderated by Electoral Safety", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Vote Share", "Treat Regional X Vote Share", "Treat District X Vote Share"),
          column.sep.width = "0pt",
          dep.var.caption="Perception of Constituent Support",
          dep.var.labels.include=FALSE,
          digits = 2,
          se=list(coef(summary(column1,cluster = c("anonid")))[, 2],
                  coef(summary(column2,cluster = c("anonid")))[, 2],
                  coef(summary(column3,cluster = c("anonid")))[, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage
## Robustness Check - State Issues Only
As a robustness check, we also analyzed the experiment using only state-level policies.

```{r, echo=FALSE, message=FALSE, results='asis'}
data.state <- subset(data.long, data.long$dv %in% c("abortion", "crime", "enviro", "gun", "transit"))
data.state.2018 <- subset(data.long.2018, data.long.2018$dv %in% c("abortion", "crime", "enviro", "gun", "transit"))
data.state.both <- bind_rows(data.state, data.state.2018)

policy1 <- lm(dv_policy ~ as.factor(treat), data = data.state)
policy1.s <- lm(dv_policy ~ as.factor(treat) + as.factor(state), data = data.state)
policy2 <- lm(dv_policy ~ as.factor(treat) + democrat + upper + trump_percent, data = data.state)
policy2.s <- lm(dv_policy ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state)
policy3 <- lm(dv_policy ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state.both)
stargazer(policy1, policy1.s, policy2, policy2.s, policy3, title="Experimental Results: Personal Policy Agreement (State Only)", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Personal Policy Agreement",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", rep(c("No", "Yes"), 2))),
          digits = 2,
          se=list(coef(summary(policy1,cluster = c("anonid")))[, 2],
                  coef(summary(policy1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(policy2,cluster = c("anonid")))[, 2],
                  coef(summary(policy2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(policy3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
vote1 <- lm(dv_vote ~ as.factor(treat), data = data.state)
vote1.s <- lm(dv_vote ~ as.factor(treat) + as.factor(state), data = data.state)
vote2 <- lm(dv_vote ~ as.factor(treat) + democrat + upper + trump_percent, data = data.state)
vote2.s <- lm(dv_vote ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state)
vote3 <- lm(dv_vote ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state.both)
stargazer(vote1, vote1.s, vote2, vote2.s, vote3, title="Experimental Results: Expected Voting Behavior (State Only)", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Expected Voting Behavior",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", c(rep(c("No", "Yes"), 2), "Yes"))),
          digits = 2,
          se=list(coef(summary(vote1,cluster = c("anonid")))[, 2],
                  coef(summary(vote1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(vote2,cluster = c("anonid")))[, 2],
                  coef(summary(vote2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(vote3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage

```{r, echo=FALSE, message=FALSE, results='asis'}
const1 <- lm(dv_const ~ as.factor(treat), data = data.state)
const1.s <- lm(dv_const ~ as.factor(treat) + as.factor(state), data = data.state)
const2 <- lm(dv_const ~ as.factor(treat) + democrat + upper + trump_percent, data = data.state)
const2.s <- lm(dv_const ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state)
const3 <- lm(dv_const ~ as.factor(treat)  + democrat + upper + trump_percent + as.factor(state), data = data.state.both)
stargazer(const1, const1.s, const2, const2.s, const3, title="Experimental Results: Perception of Constituent Support (State Only)", type = 'latex', header = FALSE, initial.zero = FALSE, keep.stat = c("n", "rsq"),
          covariate.labels = c("Treat Regional", "Treat District", "Democrat", "Upper Chamber", "Trump \\%"),
          column.sep.width = "0pt",
          dep.var.caption="Perception of Constituent Support",
          dep.var.labels.include=FALSE,
          omit="factor\\(state",
          add.lines=list(c("State FE", c(rep(c("No", "Yes"), 2), "Yes"))),
          digits = 2,
          se=list(coef(summary(const1,cluster = c("anonid")))[, 2],
                  coef(summary(const1.s,cluster = c("anonid")))[1:3, 2],
                  coef(summary(const2,cluster = c("anonid")))[, 2],
                  coef(summary(const2.s,cluster = c("anonid")))[1:6, 2],
                  coef(summary(const3,cluster = c("anonid")))[1:6, 2]),
          notes = c("Cluster-robust standard errors", "at the state legislator level."))
```

\newpage
# Additional Survey Results - Original Study

How do you normally get information about your constituents' policy preferences? Select all that apply.

Outcome | All Respondents | Complier Respondents
---------|  ----- |  ----- 
Reading the newspaper and news websites | `r report.percent(data.access, "pref_newspaper")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_newspaper")`
Social media | `r report.percent(data.access, "pref_social")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_social")`
Town halls with constituents | `r report.percent(data.access, "pref_townhall")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_townhall")`
Meetings with community leaders | `r report.percent(data.access, "pref_leaders")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_leaders")`
Phone calls, emails and letters from the constituents | `r report.percent(data.access, "pref_contact")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_contact")`
Lobbyists or other non-elected political professionals | `r report.percent(data.access, "pref_lobby")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_lobby")`
Polling | `r report.percent(data.access, "pref_polling")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_polling")`
Other legislators | `r report.percent(data.access, "pref_leg")` | `r report.percent(subset(data.access, data.access$accessed_site ==1), "pref_leg")`
N | `r nrow(data.access[!is.na(data.access[,"pref_leg"]),])` | `r nrow(data.access[!is.na(data.access[,"pref_leg"]) & data.access$accessed_site ==1 ,])`

How do you usually get your polling? Select all that apply. (among those who selected polling to the above question)

Outcome | All Respondents | Complier Respondents
---------|  ----- |  ----- 
I commission it through my campaign | `r report.percent(data.access, "polling_campaign")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "polling_campaign")`
I commission it through my legislative office | `r report.percent(data.access, "polling_office")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "polling_office")`
I rely on state party polling | `r report.percent(data.access, "polling_state_party")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "polling_state_party")`
I rely on national party polling | `r report.percent(data.access, "polling_national_party")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "polling_national_party")`
I rely on polling described in the media | `r report.percent(data.access, "polling_media")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "polling_media")`
N | `r nrow(data.access[!is.na(data.access[,"polling_media"]),])` | `r nrow(data.access[!is.na(data.access[,"polling_media"]) & data.access$accessed_site ==1 ,])`

If you had the opportunity, would you make use of polling about the policy preferences of your constituents? 

Outcome | All Respondents | Complier Respondents
---------|  ----- |  ----- 
Yes, if it were complimentary | `r report.percent(data.access, "want_free")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "want_free")`
Yes, even if I had to pay for it | `r report.percent(data.access, "want_pay")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "want_pay")`
No, no matter what | `r report.percent(data.access, "want_no")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "want_no")`
I'm not sure | `r report.percent(data.access, "want_idk")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "want_idk")`
N | `r nrow(data.access[!is.na(data.access[,"want_idk"]),])` | `r nrow(data.access[!is.na(data.access[,"want_idk"]) & data.access$accessed_site ==1 ,])`

If you had polling information about your constituents, what would you be more likely to do with it in most situations? Please select the option that you think best reflects what you would be more likely to do. 

Outcome | All Respondents | Complier Respondents
---------------------------|  -------------- |  --------------
I would work to change the preferences of my constituents. | `r report.percent(data.access, "preferences_change")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "preferences_change")`
I would work to better follow the preferences of my constituents. | `r report.percent(data.access, "preferences_follow")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "preferences_follow")`
N | `r nrow(data.access[!is.na(data.access[,"preferences_follow"]),])` | `r nrow(data.access[!is.na(data.access[,"preferences_follow"]) & data.access$accessed_site ==1 ,])`

Do you recall receiving polling information about your district from District Pulse, an organization led by political scientists **REMOVED**?

Outcome | All Respondents | Complier Respondents
---------------------------|  -------------- |  --------------
I remember | `r report.percent(data.access, "recall_remember")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "recall_remember")`
I don't remember | `r report.percent(data.access, "recall_dontremember")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "recall_dontremember")`
I'm not sure | `r report.percent(data.access, "recall_idk")` | `r report.percent(subset(data.access, data.access$accessed_site == 1), "recall_idk")`
N  | `r nrow(data.access[!is.na(data.access[,"recall_idk"]),])` | `r nrow(data.access[!is.na(data.access[,"recall_idk"]) & data.access$accessed_site ==1 ,])`

# SM References

Ahler, Douglas J. and David E. Broockman. 2017. "The Delegate Paradox: Why Polarized Politicians Can Represent Citizens Best." Journal of Politics . (forthcoming).

Broockman, David, and Christopher Skovron. 2018. "Bias in perceptions of public opinion among american political elites". American Political Science Review. (forthcoming).

Broockman, David, Joshua L. Kalla, and Jasjeet S. Sekhon. 2017. "The design of field experiments with survey outcomes: A framework for selecting more efficient, robust, and ethical designs". Political Analysis 25(4), 435-464.

Broockman, David, Nicholas Carnes, Melody Crowder-Meyer, and Christopher Skovron. 2017. "Having Their Cake and Eating It, Too: (Why) Local Party Leaders Prefer Nominating Extreme Candidates." Working Paper

Green, Donald P., Anna Wilke, and Jasper Cooper. 2017. "Reducing Intimate Partner Violence through Informal Social Control: A mass media experiment in rural Uganda". Working Paper. 

Higgins, Michael. J., Fredrik Sävje, and Jasjeet S. Sekhon. 2016. "Improving massive experiments with threshold blocking. Proceedings of the National Academy of Sciences". 113(27), 7369-7376.

Teele, Dawn L., Joshua L. Kalla, and Francess McCall Rosenbluth. 2018. "The Ties that Double Bind: Social Roles and Women's Underrepresentation in Politics". Working Paper. 


